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Abstract 

Two experiments were conducted to assess an online version of Keller’s personalized system of Instruction, called computer- 
aided personalized system of instruction (CAPSI), as part of a blended learning design with regard to course knowledge and 
critical thinking development. In Experiment 1, two lecture sections of an Introduction to University course received a CAPSI 
assignment while two received an extra paper assignment. In Experiment 2, one lecture section of an Introduction to University 
course received a CAPSI assignment while another lecture section was assigned a research paper. In the two experiments 
the CAPSI sections consistently outperformed the sections with which they were compared, indicating that CAPSI is a viable 
option in higher education. 
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B ehavior analysis arguably has played a major role in the 
movement away from the traditional approach to higher 
education (B. F. Skinner, 1958,1968; see Pascarella & Teren- 
zini, 1991, 2005). One widely recognized behavioral approach 
to higher education is Kellers (1968) personalized system of in¬ 
struction (PSI). The major components of PSI are: (a) instructor- 
selected textbooks and other textual material; (b) study ques¬ 
tions based on the assigned material, (c) student self-pacing; (b) 
clear study objectives; (c) stress on the written word; (d) textual 
material divided into small study units; (e) demonstration of 
mastery of one study unit of material before proceeding to the 
next study unit; (f) use of more advanced students (typically in 
another course) to administer unit tests and provide immedi¬ 
ate feedback; and (g) use of lectures for motivational purposes 
as opposed to providing new information. In a PSI approach, 
the instructor taking on the role of facilitator allows students to 
become more interactive with the material. 

PSI has been shown to be more effective than the traditional 
lecture method in developing student learning (Kulik, Kulik, & 
Bangert-Downs, 1990; Kulik, Kulik, & Cohen, 1979). Students 
in PSI-taught courses have achieved higher exam scores with 
less instructional time, and professed greater enjoyment toward 
learning the course material. Differences were more pronounced 
when performance was measured on essay exams as opposed to 
multiple-choice exams. 

Computer-aided personalized system of instruction (CAPSI) 
is a computerized form of PSI that has evolved from a mainframe 
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to a Web-based environment (Kinsner & Pear, 1988, 1990; Pear, 
2002; Pear & Crone-Todd, 1999, 2002; Pear & Kinsner, 1988; 
Pear & Martin, 2004; Pear & Novak, 1996; Pear, Schnerch, Silva, 
Svenningsen, & Lambert, in press; Springer & Pear, 2008). In ad¬ 
dition to being delivered by computer, CAPSI differs from tra¬ 
ditional PSI in that rather than having more advanced students 
outside of the course assess and provide feedback on unit tests, 
CAPSI enables more advanced students (called peer reviewers) 
within a course to perform this function. Thus, once a student 
demonstrates mastery on a given unit, that student may serve as 
a peer reviewer for that unit. Unit tests must be assigned a pass 
either by two peer reviewers, by the instructor, or by a teaching 
assistant in order for the student to proceed to the next unit. If a 
student does not pass a unit test, the student may try again on a 
new test of the unit after a period of restudy or appeal the result. 
The appeal is submitted through the program to the instruc¬ 
tor, who then judges the strength of the student’s argument and 
decides whether or not to accept the appeal. CAPSI allows stu¬ 
dents to select the time (24-hour availability) at which they are 
ready to write a unit test, based on their personal judgment that 
they have mastered the material in the unit. The result of a unit 
test indicates whether the student is correct in this judgment or 
whether restudying the unit is necessary. Since unit tests are es¬ 
say and short-answer rather than multiple choice, and students 
must demonstrate mastery on each study unit before proceeding 
to the next unit, they cannot progress in a course without under¬ 
standing the previous required material. This helps ensure that 
the instructional goals are always appropriate to a student’s level 
of understanding of the course material. 
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The combination of classroom instruction with Web-based 
learning is known as blended learning (Alonso, Lopez, Man- 
rique, & Vines, 2005; Carr-Chellman & Duchastel, 2001; Con- 
cannon, Flynn, & Campbell, 2005; Dziuban, Hartman, & Mos- 
kal, 2007; Merisotis, 2001). Although CAPSI can be used as a 
stand-alone method, it is also compatible with a blended ap¬ 
proach. Students can take the CAPSI unit tests on their own 
schedules, collaboration can occur between students and be¬ 
tween students and the instructor both in class and on the Inter¬ 
net, assessment can occur from both in-class quizzes and exams 
and the CAPSI unit tests, while performance-support materials 
can be provided within the CAPSI program or from other ma¬ 
terials, such as textbooks. 

CAPSI was designed to foster students’ critical or higher- 
order thinking (e.g., Crone-Todd & Pear, 2001; Crone-Todd , 
Pear, & Read, 2000; Pear, Crone-Todd, Wirth, & Simister, 2001). 
There are many definitions of critical thinking, although not all 
sound behavioral. Skinner perhaps came close to providing a 
definition in his discussion of thinking as verbal behavior that 
occurs in a speaker who is also his or her own listener (Skinner, 
1957, pp. 438-452). This aspect of critical thinking is captured 
in the following definition arrived at by a panel of educational 
experts: 

We understand critical thinking to be purposeful, self-reg¬ 
ulatory judgment which results in interpretation, analysis, 
evaluation, and inference, as well as explanation of the evi¬ 
dential, conceptual, methodological, criteriological, or con¬ 
textual considerations upon which that judgment is based” 
(Facione, 1990, p. 3). 

It has been argued, however, that although PSI, and by impli¬ 
cation CAPSI, may be effective for basic knowledge learning, 
it cannot develop critical thinking skills (Adams, 2004; Budd, 
2002). In response, Grant and Spencer (2003) pointed out that 
any criticism of PSI in regard to teaching critical thinking is 
equally applicable to university courses in general. More specifi¬ 
cally, Reboy and Semb (1991) made the following four points: 
First, PSI is a delivery system that is not dependent on content; 
therefore, content within the PSI delivery system can be focused 
on developing critical thinking skills by using a systematic ques¬ 
tioning technique (Quitadamo & Brown, 2001). Second, PSI 
has been successfully used in courses requiring critical think¬ 
ing skills. Third, research has shown that “students enrolled in 
PSI courses have improved their higher-order cognitive [think¬ 
ing] skills” (p. 212). Fourth, PSI allows for “early diagnosis of 
inadequate reasoning because of frequent student contact and 
assessment” (p. 213). 

The emphasis in CAPSI on written answers, providing writ¬ 
ten feedback when assessing other students’ answers, and writ¬ 
ing appeals develops critical thinking as discussed above (i.e., 
engaging in verbal behavior under the control of the verbal be¬ 
havior of oneself as well as that of others). Teaching by comput¬ 
er-mediated communications encourages interacting through 
written forms of communication, which it has been argued 
may result in increased reflection and greater problem-solv¬ 
ing performance (Garrison & Kanuka, 2004; King, 1995). The 
range and quality of interactive dialogue that can be facilitated 
through CAPSI is consistent with this view. 


The present study sought to examine in two experiments con¬ 
ducted in two courses the pedagogical effectiveness of CAPSI 
as a component of a blended course with respect to students 
learning course material and developing critical thinking skills. 

■ EXPERIMENT 1_ 

Introduction to University at the University of Manitoba is a 
first year university course that introduces students to the skills 
they will need for other university courses, thus helping stu¬ 
dents make the transition from high school to university. The 
course normally requires students to write three essay assign¬ 
ments. The hypothesis in Experiment 1 was that completing 
a CAPSI assignment develops course knowledge and critical 
thinking skills above that of writing a third essay assignment. 

This hypothesis was tested using the final examination and 
a critical thinking assessment tool called the Applied Critical 
Thinking Measure (ACTM). The final exam consisted of three 
types of questions exemplifying three levels of Bloom’s taxon¬ 
omy (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956): (a) 
knowledge of the course textual material; (b) ability to apply 
either problem solving skills or to illustrate their knowledge of 
Bloom’s taxonomy; and (c) use of Hegel’s Dialectic as evidence 
of critical thinking. 

■ METHOD 


PARTICIPANTS 

The participants were 364 University of Manitoba students en¬ 
rolled in Introduction to University for the Fall 2006 13-week 
academic term. The course was only open to students who had 
completed less than twelve credit hours. 

PROCEDURE 

The course had four lecture sections: two CAPSI sections and 
two non-CAPSI sections. Eight lab sections accompanied each 
lecture section. The sections were comparable because students 
signed up for particular lecture sections and corresponding lab 
sections without prior knowledge of the methods to be used in 
the different lecture sections. 

Course requirements were similar across all four sections. 
The only difference was that students in the two CAPSI sections 
were required to complete two APA-formatted essays (roughly 
1000 words each) and a CAPSI assignment, while students in 
the two the non-CAPSI sections were required to write three 
APA-formatted essays (roughly 1000 words each) as a part of 
course requirements 

Four different instructors were responsible for each of the 
four lecture sections of the course. Each instructor delivered 
the in-class lecture component and supervised eight teaching 
assistants (TAs) for the eight labs associated with each lecture 
section. For the two CAPSI sections, one instructor was the 
first author; the other instructor had previous knowledge of the 
CAPSI program. All instructors had a minimum of three years 
experience teaching the course, with the instructors of the non- 
CAPSI sections having greater experience teaching the course. 

Attending regularly scheduled instructor meetings was a re¬ 
quirement for teaching the course. One component of these 
meetings was to insure that all instructors were aware of the 
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general objectives and the teaching and learning expectations 
of the course. This approach increased the teaching consistency 
across the various sections to insure that all students received 
the same level of instruction. 

TAs were assigned on the basis of their availability to their 
respective lab sections. During the labs the TAs led group ex¬ 
ercises, facilitated individual presentations, and assisted in the 
development of writing skills. The TAs also graded essay assign¬ 
ments, the final exam, and, in the CAPSI sections, unit tests that 
were not assessed by peer reviewers. Unit tests were graded on 
a mastery basis. For mastery to be achieved, the peer reviewers 
or TAs determined that three questions related to the unit were 
answered with 100% accuracy. 

Tectures were 75 minutes long and occurred once a week on 
either Tuesdays or Thursdays, depending on the lecture section. 
Labs were 75 minutes long and occurred once a week on the 
Tuesday or Thursday that was not being used for its associated 
lecture. 

All lecture sections used the same textbook: Study and Criti¬ 
cal Thinking Skills in College (Sixth Edition) by Kathleen Mc¬ 
Whorter (2006). Students were also given a course outline, 
which detailed the course requirements. Students in the CAPSI 
sections received information about the CAPSI assignment, 
while students in the non-CAPSI sections received information 
about writing a third essay. 

All students wrote the same final exam in the same exam pe¬ 
riod after lectures had ended. The final exam consisted of three 
components: (a) a knowledge component; (b) an applied com¬ 
ponent; and (c) a critical thinking component. 

THE CAPSI PROGRAM 

In a lecture session during the second week of classes the stu¬ 
dents in the CAPSI sections were shown and directed to the 
CAPSI website at www.capsi.org. At the same time, a demon¬ 
stration of CAPSI was conducted to show students how to move 
through the system to take unit tests and to peer review other 
students’ unit tests. 

Prior to the beginning of the course, the instructors for the 
two CAPSI sections gave the TAs in their sections a demonstra¬ 
tion of the CAPSI program, with instructions regarding proper 
feedback procedures. In addition, both instructors maintained 
weekly contact with the TAs either by email or in person in or¬ 
der to address any concerns and to provide general feedback 
and motivation. 

The CAPSI program required that unit tests be graded by 
either a TA or by two peer reviewers. When assigning peer 
reviewers, the CAPSI program selected two peers with the 
lowest number of peer-reviewing points who had passed the 
unit being graded and who had agreed to be available to grade 
within 24 hours after the unit test to be graded was submitted. 
When two peer reviewers assessed a unit test, both had to inde¬ 
pendently agree that the student demonstrated mastery of the 
material in order for the program to record the test as a pass. 
All peer reviewing was anonymous with respect to both the test 
writer and the peer reviewer. There was a built-in appeal process 
for arguing the validity of a given answer. 

The CAPSI assignment was designed for students to pass 20 
unit tests and to peer review 10 unit tests. Each CAPSI unit 


corresponded to one chapter from the course textbook. Each 
unit test was worth .75 % of the final grade, while each peer 
review was worth .5%. For each unit, students were required to 
answer 3 randomly generated questions from the CAPSI unit 
corresponding to the appropriate chapter from the textbook. 
The questions for each unit were composed according to the 
requirements for the six levels of Bloom’s taxonomy and were 
designed to test the students’ understanding of the textbook 
material. 

THE APPLIED CRITICAL THINKING MEASURE (ACTM) 

The ACTM and the associated scoring sheet were designed and 
researched by Dr. Robert Renaud of the Education Faculty at 
the University of Manitoba. (Regehr, 2003; Renaud & Mandzuk, 
2006). Although the psychometric properties of the ACTM have 
not been extensively studied, there were two reasons for choos¬ 
ing it as the critical thinking measure. First, both CAPSI study 
unit questions and the ACTM items implement a question-an¬ 
swer method in which the answers are in a written as opposed 
to a multiple-choice format. Second, the ACTM is consistent 
with the view of critical thinking discussed earlier in this paper. 

The ACTM was administered to students attending the last 
lab at the end of the term. There was no requirement that stu¬ 
dents had to attend the lab or participate in writing the ACTM. 
Therefore, the number of students writing the ACTM was lower 
than the number of students actually writing the exam. 

In this measure, students were asked to respond to three 
vague situations they were likely to be familiar with. They were 
then asked to make a “Yes,” “No,” or “Not sure” selection based 
on what they had read and the question that was asked at the 
end of the scenario. Students were then directed to either write 
three statements that justified their “Yes” or “No” decision; or, 
if “Not Sure” was selected, to specify three questions regarding 
what information they would need in order to make a justified 
decision. 

A grading rubric was supplied to assessors to rate the re¬ 
sponses to the ACTM scenarios. Each written response was 
rated on a scale from 0 to 1 for a “Yes” choice, 0 to 2 for a “No” 
choice, and 0 to 3 for a “Not Sure” choice. The “Not Sure” re¬ 
sponses were given the higher rating because the vague nature 
of the scenario opened it to greater critical analysis and ques¬ 
tions. Therefore, a student could obtain a critical thinking score 
from 0 to 9 points on each scenario, achieving a total score from 
0-27 for the complete assessment. A sample scenario is as fol¬ 
lows: 

According to a recent report, in Canadian prisons, the sentences are too 
short and the time spent behind bars is too easy. In fact, a considerable 
number of those convicted of violent crimes are released before they 
have served even a third of their expected sentences. 

Do you agree that the prison system in Canada is far too soft, especially 
with more serious offenders? 

(circle one) YES NO NOT SURE 

In the boxes below, if you answered YES or NO, please give three dis¬ 
tinct reasons why (one reason per box). If you answered NOT SURE, 
what would be the three most important or relevant things (one point 
per box) you want to know before deciding whether or not the prison 
system in Canada is too soft? 
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DATA COLLECTION AND RELIABILITY MEASUREMENT 

The TAs did the initial scoring of the final exam for the students 
in their respective lab sections using a grading rubric provid¬ 
ed by the instructor. To assess the reliability of the final exam 
grades a random sample of 20 final exams was re-scored by an 
independent grader using the same rubric that the TAs used. 
The independent grader was blind with regard to the lecture 
sections the papers were from. The formula used to assess reli¬ 
ability was the lower score divided by the higher score. If the 
scores were the same, then the IOR was 1.00 for that assessment. 
Using this measure a mean reliability score of .85 was obtained. 

The ACTMs were scored by the first author and two inde¬ 
pendent reliability scorers who were blind as to the lecture sec¬ 
tions that the ACTMs were associated with. The total number of 
ACTMs used for the post ACTM statistical analysis was 132; of 
these, 45 were rescored, resulting in 34.09% of the assessments 
being rescored. The formula used for the reliability measure was 
the lower score divided score by the higher score. If the scores 
were the same, the reliability measure was 1.00 for that assess¬ 
ment. Using this measure a mean reliability score of .82 was ob¬ 
tained. 

DATA ANALYSIS 

The data were analyzed using an analysis of variance (ANOVA) 
instructors-nested-within-teaching-method design. In this de¬ 
sign, students were nested within instructors and the instruc¬ 
tors were nested within the teaching methods. Four compari¬ 
sons between the CAPSI and non-CAPSI conditions were made 
in this experiment and will be presented in the results sections 
in this order: (a) total scores on the final exam; (b) scores on 
the critical thinking question of the final exam; (c) scores on 
the two content questions of the final exam; and (d) scores on 
the ACTM. 

■ RESULTS 


FINAL EXAM 

The CAPSI sections scored higher than the non-CAPSI sections 
on the final exam. The overall mean of the CAPSI sections was 
69.18 (s.d. = 6.46) and that of the non-CAPSI sections was 65.95 
(s.d. = 6.35), resulting in a difference of 3.23 in the final exam 
score between the CAPSI and non-CAPSI conditions (p = .053). 
The effect size as indexed by a partial eta-squared would be clas¬ 
sified as minimal (partial rf = .01; Tabachnick & Fidell, 2007). 

CRITICAL THINKING EXAM QUESTION 

The overall mean of the CAPSI sections was higher (M = 28.25, 
s.d. = 6.46) than the non-CAPSI sections (M = 26.74, s.d. = 6.35) 
resulting in a difference of 1.51 between the conditions. The 
mean difference between the two conditions for the critical 
thinking question was significant at the .05 level (p = .026). Al¬ 
though this difference was significant at the .05 level, the effect 
size would be classified as small (Partial rf = .01; Tabachnick & 
Fidell, 2007). 

QUESTIONS ON COURSE CONTENT 

The CAPSI sections scored higher than the non-CAPSI sections 
on the course content questions (M = 42.82 vs. 40.26). However, 
this difference was not statistically significant (the s.d.’s were 


22.94 and 17.99, respectively). 

ACTM 

The CAPSI sections scored higher than the non-CAPSI sections 
on the ACTM (M = 9.33 vs. 8.62). However, this difference was 
not statistically significant (the s.d.’s were 5.13 and 6.18, respec¬ 
tively). 

■ DISCUSSION_ 

The major finding of Experiment 1 was that students in the 
CAPSI sections scored higher on the final exam than students 
in the non-CAPSI sections. This appears to be due primarily 
to the performance on the critical thinking question, because 
there was a significant difference between the conditions on that 
question, but not on the content questions. Although there were 
non-significant differences between the CAPSI and the non- 
CAPSI sections on the content questions and the ACTM, these 
differences did favor the CAPSI condition. 

The results support earlier findings on the effectiveness of 
PSI (Kulik et al., 1979, 1990; M. Skinner, 1990). The results also 
support the findings that students in blended learning courses 
perform as well or better than students in strictly lecture-based 
courses (Means, Toyama, Murphy, Bakia, & Jones, 2009; Pereira, 
Pleguezuelos, Molina-Ros, Molina-Tomas, & Masdeu, 2007). 

Although the main differences favored the CAPSI condition, 
the effect sizes were small. This contrasts with the large effect 
sizes that have been obtained in comparisons of PSI with the 
lecture method. Possible reasons for this discrepancy will be ad¬ 
dressed in the General Discussion section. With regard to the 
lack of a significant effect on the ACTM scores, it should be 
noted the power of the statistical test could have been decreased 
in three ways. First, there were fewer participants who wrote 
the ACTM than who wrote the final exam. Second, conducting 
the ACTM in the labs could have increased the variability of 
the scores in that the TAs may have provided different instruc¬ 
tions to the students, or provided different responses to ques¬ 
tions from the students. Third, it is possible that the ACTM was 
not sensitive enough to pick up some of the subtle effects that 
CAPSI may have had on developing critical thinking. 

■ EXPERIMENT 2_ 

The high variability between instructors in Experiment 1 may 
have tended to mask differences between the CAPSI and non- 
CAPSI conditions. Therefore, to eliminate variability between 
instructors, Experiment 2 compared the CAPSI and non-CAP¬ 
SI treatments in two lecture sections of Introduction to Uni¬ 
versity that were both taught by the same instructor. In addi¬ 
tion, whereas Experiment 1 examined both the development 
of knowledge and critical thinking, Experiment 2 focused ex¬ 
clusively on the latter. To this end, the CAPSI questions were 
only of the highest level - Blooms level 6. That is, the CAPSI 
component used only questions that required students to evalu¬ 
ate a position supported by information from the course text¬ 
book. Other differences with Experiment 1 are detailed below. 
The hypothesis to be tested was that student participation in a 
CAPSI assignment in which students are asked to supply higher 
level thinking responses as a component of a blended designed 
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course develops student critical thinking more effectively than 
writing a third essay. 

■ METHOD 

PARTICIPANTS 

The participants were 276 University of Manitoba students en¬ 
rolled in two lecture sections of Introduction to University for 
the Fall 2007 13-week academic term. 

PROCEDURE 

One lecture section was designated the CAPSI section and the 
other the non-CAPSI section. The two lecture sections were 
scheduled for either Tuesdays or Thursdays. Through a simple 
random method of flipping a coin and having a third party call 
the toss, it was determined that the Thursday lecture section 
would receive the CAPSI assignment. 

The instructor in this experiment was Instructor 1 from 
Experiment 1. Eight TAs were selected by the department in 
charge of the course and matched with lab sections based on 
availability. There were eight lab sections associated with each 
of the two lecture sections, with each lab section having a maxi¬ 
mum enrolment of 20 students. The labs were scheduled once 
a week on the Tuesday or Thursday that was not being used for 
the associated lecture. 

The number of students originally registered in the CAPSI 
section was 137 and the number of students originally regis¬ 
tered in the non-CAPSI section was 139. As in Experiment 1, 
students signed up for the lecture sections without prior knowl¬ 
edge of the teaching methods used in the different lecture sec¬ 
tions. 

The textbook used for the course was ARTS 1110 Introduc¬ 
tion to University: Custom Edition for the University of Mani¬ 
toba (2007). The book was a compilation of chapters from Mc- 
Whorther (2006) and Browne and Keeley (2007). All students 
used the same textbook and were given the same course outline. 

All students were required to write two 1,000-word APA- 
style essays. However, students in the non-CAPSI section were 
required to write a third 1,000-word APA-styled essay while 
students in the CAPSI section had CAPSI as a required assign¬ 
ment. In addition, students in both lecture sections were given 
an ACTM prior to the beginning of the CAPSI assignment and 
at the end of the term. Different questions were used in the two 
administrations of the ACTM. This was done to avoid a practice 
effect from writing answers to the same questions in the pre- 
ACTM assessment. The CAPSI program was set up in a similar 
manner to that of Experiment 1. 

CAPSI ASSIGNMENT 

In Experiment 1, the CAPSI program amalgamated all students 
in the CAPSI sections and all TAs as part of the same large class. 
In Experiment 2, the CAPSI program treated the eight lab sec¬ 
tions as separate classes. That is, each lab section was a separate 
CAPSI class. TAs only assessed answers from students in their 
lab section; and students only peer reviewed other students in 
their lab section. The reasons for this were: (a) to increase the 
efficiency of certain aspects of the program and (b) to build a 
greater sense of responsibility and communication within the 


sections. Students could ask the TA directly about progress 
through CAPSI or other specific questions related to the assign¬ 
ment. 

Similar procedures were followed regarding TA training. The 
only changes being first, that instruction was provided just prior 
to the beginning of the CAPSI assignment, not the beginning of 
the course. Second, only the one instructor maintained weekly 
contact with the TAs either by email or in person in order to 
address any concerns and to provide general feedback and mo¬ 
tivation. 

Whereas in Experiment 1, students could access the program 
from the first week of classes, in Experiment 2, students were 
able to take unit tests only in the last third of the academic term. 
The purpose of this was to correspond to the timing of the re¬ 
quired third essay assignment of the non-CAPSI section and to 
the material that was presented in the lectures. As in Experiment 
1, in the week prior to the beginning of the CAPSI assignment, 
students in the CAPSI section were shown and directed to the 
CAPSI website and given a demonstration of the program. 

The design of the questions for the CAPSI assignment fol¬ 
lowed J. M. Keller’s (1999) model of attention, relevance, confi¬ 
dence, and satisfaction (ARCS). In following this model, small 
modules were set up to help students master one unit at a time, 
provide students with clear criteria, and deliver effective and 
timely feedback for motivation and confidence. 

Only the final eight chapters of the textbook were used. These 
were the chapters that focused on the aspect of critical think¬ 
ing and critical assessment. These eight chapters were then con¬ 
densed into five units. 

The CAPSI assignment was designed for students to pass 5 
unit tests and to peer review 5 times. Each unit test consisted 
of 1 question randomly selected by the CAPSI program from a 
database of questions. Each question was worth 2% of the final 
grade and each instance of peer reviewing was worth 1%. There¬ 
fore, if a student completed all 5 unit tests and peer reviews, the 
student would have earned 15% of his or her grade. A student 
received a final grade on the assignment associated with the 
number of units passed and peer reviews completed. 

The questions for each unit were designed to correspond to 
the appropriate chapters from the course textbook. The answers 
could not be found in the chapters per se; however, information 
from the chapters could be incorporated into the answers. The 
instructor originally selected the questions, which were then 
submitted to the first author for approval prior to inclusion in 
the CAPSI program. The questions were either taken from let¬ 
ters sent into a daily newspaper or were designed by the instruc¬ 
tor for the course. All references to the source of the questions, 
as well as authors of the letters were removed from the original 
letter. 

DATA COLLECTION AND ASSESSMENT 

Students in both lecture sections were given a pre-ACTM prior 
to the beginning of the CAPSI assignment in the CAPSI section 
or third essay assignment in the non-CAPSI section. The ACTM 
was administered to both lecture sections in the last week of 
classes. The pre-ACTM contained problems similar to, but dif¬ 
ferent from those the students received on the ACTM at the end 
of the term. The same grading rubric as in Experiment 1 was 
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used to assess the responses to the ACTM scenarios. Three criti¬ 
cal thinking questions were assigned in the Pre-ACTM, while a 
different set of three questions assigned in the Post-ACTM. The 
questions used on the Pre-ACTM and ACTM were the same 
ones that were used in Experiment 1. 

INTER-RATER RELIABILITIES 

Reliability of assessing the Pre-ACTM and the ACTM were cal¬ 
culated in the same manner as in Experiment 1. The reliability 
measures obtained were .74 and .83, respectively. 

■ RESULTS_ 

The mean pre-ACTM score for the CAPSI section was only 
slightly higher than the non-CAPSI section (3.83 vs. 3.81, 
s.d.’s = 1.94 and 1.80, respectively), indicating that the two lec¬ 
ture sections were approximately equal in critical thinking prior 
to the CAPSI assignment. However, the difference between the 
CAPSI and non-CAPSI sections increased on the ACTM by a 
factor of over 41 (.0174 vs. .7295; s.d.’s = 1.79 and 1.96, respec¬ 
tively). 

The scores on the pre-ACTM did not differ significantly 
between the two lecture sections. However, the scores on the 
ACTM differed significantly (p = .03) using a two-tailed test. 
The effect size as indexed by Cohen’s d was .389, which would 
be classified as medium (Cohen, 1988, p. 25). 

■ DISCUSSION_ 

The results indicate that the CAPSI program was effective in 
increasing critical thinking skills as measured by the ACTM. 
The results on the pre-ACTM indicate that the students in both 
lecture sections were at approximately the same level of critical 
thinking ability at the start of the CAPSI assignment or third 
essay assignment. While students in both lecture sections did 
show an increase in critical thinking ability, the CAPSI section 
showed greater improvement than the non-CAPSI section. 

This result supports Grant and Spencer’s (2003) point that 
PSI is applicable to teaching critical thinking in post-secondary 
courses. The result is also consistent with Reboy and Semb’s 
(1991) finding of improved higher-order thinking skills in stu¬ 
dents enrolled in PSI courses. 

■ GENERAL DISCUSSION_ 

The positive outcomes in both experiments indicate that CAPSI 
is an effective component for developing critical thinking and 
course knowledge in blended designs. Two general reasons why 
the effects observed may not have been stronger are as follows. 
First, as Terenzini, Springer, Pascarella, and Nora (1995) pointed 
out, “one possible explanation for the absence of significant ef¬ 
fects [in comparing differences between instructional methods 
in university courses] may be that a semester course experience 
maybe too brief to produce any measurable impact” (p. 24). In 
other words, the differences observed in this experiment might 
have been more pronounced in a longer course in which CAPSI 
was used for a longer period of time. Second, not all students in 
the CAPSI conditions completed the unit tests. Therefore, the 
consistently positive results of the CAPSI conditions were based 
on a smaller amount of exposure to the CAPSI component than 


would be indicated by a superficial look at the procedure. 

We now discuss some specific effects that with regard to de¬ 
veloping critical thinking and learning course content. 

DEVELOPING CRITICAL THINKING 

Contrary to the criticism that behaviorist methods are not ef¬ 
fective for developing higher level thinking skills (Budd, 2002), 
scores on critical thinking measures showed a consistently posi¬ 
tive difference in favor of the CAPSI sections in the two experi¬ 
ments. The results from these experiments add to the findings 
of Reboy and Semb (1991) that PSI can be used to teach critical 
thinking at levels of achievement that exceed those attained in a 
lecture-discussion format. As stated by Reboy and Semb (1991), 
the positive effect of PSI on critical thinking could be due to 
early diagnosis of inadequate answers that increases the prob¬ 
ability of detecting and correcting errors in reasoning. The early 
diagnosis and correction in CAPSI is attained by receiving feed¬ 
back on written answers and on appeals that argue the merits of 
specific answers. 

Other factors that could have contributed to this result are 
the types of questions asked, especially in Experiment 2, and 
the inclusion of peer reviewing. As stated earlier, Reboy and 
Semb (1991) pointed out that the components of the PSI deliv¬ 
ery system can be focused on developing higher-order thinking 
skills. That peer reviewing might be a factor contributing to the 
development of critical thinking is consistent with Terenzini et 
al. (1995), who stated that interactions among students seems 
to have a positive influence on critical thinking. For CAPSI, the 
interactions involving peer reviewing could have produced a 
similar enhancement of critical thinking. Peer reviewing can be 
viewed as a highly systematic written dialogue between students 
on specific course information (Carr-Chellman & Duchastel, 
2001 ). 

LEARNING COURSE CONTENT 

The consistent superiority of CAPSI is in accordance with the 
findings of greater college-level exam scores for PSI type learn¬ 
ing (Kulik et al., 1979,1990; M. Skinner, 1990). It is also consis¬ 
tent with the findings of Dzubian et al. (2004), and Pereira et al. 
(2007) who showed that students in blended learning courses 
achieved higher scores than students in lecture-based courses. 
The small effect sizes found in the present study are not con¬ 
sistent with the larger effect sizes found in previous studies on 
PSI. However, most of the prior studies compared PSI with a 
straight lecture method. In the present study, steps were taken to 
ensure that the methods used in the comparison sections were 
as effective as possible; e.g., assigning essays, and providing TAs 
that assisted students in their learning . In addition, in both ex¬ 
periments, CAPSI was blended with lectures that were identical 
across sections. That the CAPSI sections still consistently out¬ 
performed the sections with which they were compared with in 
regard to course content learning and critical thinking supports 
the view that CAPSI is a highly effective empirically based edu¬ 
cational methodology. 
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